<html>
<head><meta charset="utf-8"><title>Linker pipelining · t-compiler/wg-pipelining · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/index.html">t-compiler/wg-pipelining</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Linker.20pipelining.html">Linker pipelining</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="174939907"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Linker%20pipelining/near/174939907" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Linker.20pipelining.html#174939907">(Sep 05 2019 at 02:39)</a>:</h4>
<p>One thing I talked with others at rustconf a bit is splitting out a linker step from rustc, and while motivated by a bunch of other things it's also pretty relevant to pipelining. Often the final crate can be one of the biggest (think build scripts, procedural macros, binaries, etc), but we currently can't pipeline them at all. If Cargo could, however, split out the linker step, we would be able to pipeline everything.</p>
<p>Imagine, for example, that instead of <code>rustc foo.rs --tons --of --flags</code> Cargo instead executed:</p>
<div class="codehilite"><pre><span></span>$ rustc foo.rs --do-not-link --tons --of --flags
$ rustc foo.rs --do-only-link --tons --of --flags
</pre></div>


<p>If we had a scheme like that then <code>--do-not-link</code> would be a pipelineable compilation (it only needs the rmeta from upstream crates) and <code>--do-only-link</code> would have to wait for everything to fully finish. This would be a relatively large-ish change to the compiler but perhaps not the worst thing in the world. Ideally <code>--do-only-link</code> would be <em>very</em> fast since it would forcibly skip all of typechecking and such. Ideally <code>--do-not-link</code> would emit something like a <code>*.rlib</code> but perhaps call it a <code>*.rlink</code> which has all the intermediate information for <code>--do-only-link</code> to immediately resume at the linking phase.</p>
<p>I think this would allow us to get 100% of the benefit from pipelining architecturally (other than making rmeta generation cheaper and further forward in compilation). This would also have lots of cascading effects on other strategies which want to separate the linker. This would be a relatively big feature for the compiler, though, so it probably wants both experimentation and an RFC to accompany it. Wanted to make sure I wrote down the thoughts though!</p>



<a name="175010895"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Linker%20pipelining/near/175010895" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Linker.20pipelining.html#175010895">(Sep 05 2019 at 20:17)</a>:</h4>
<p>I've <a href="https://github.com/rust-lang/rust/issues/64191" target="_blank" title="https://github.com/rust-lang/rust/issues/64191">opened an issue</a> for this topic</p>



<a name="204258226"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Linker%20pipelining/near/204258226" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Linker.20pipelining.html#204258226">(Jul 17 2020 at 20:58)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> Has there been any progress on linker pipelining? Is there enough that Cargo could add preliminary support?</p>



<a name="204258315"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Linker%20pipelining/near/204258315" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Linker.20pipelining.html#204258315">(Jul 17 2020 at 20:59)</a>:</h4>
<p><span class="user-mention" data-user-id="239881">@Josh Triplett</span> I'm not aware of progress myself</p>



<a name="204258337"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Linker%20pipelining/near/204258337" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Linker.20pipelining.html#204258337">(Jul 17 2020 at 20:59)</a>:</h4>
<p>I remember a PR for <code>--only-link</code> but I'm not sure where that PR ended up</p>



<a name="204260758"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Linker%20pipelining/near/204260758" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Linker.20pipelining.html#204260758">(Jul 17 2020 at 21:20)</a>:</h4>
<p>I saw the one for not linking; I didn't see the one for only linking.</p>



<a name="204260787"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Linker%20pipelining/near/204260787" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Josh Triplett <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Linker.20pipelining.html#204260787">(Jul 17 2020 at 21:21)</a>:</h4>
<p>I'm very interested in anything that will allow more parallelism. :)</p>



<a name="204265906"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Linker%20pipelining/near/204265906" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Linker.20pipelining.html#204265906">(Jul 17 2020 at 22:15)</a>:</h4>
<p>one of the neat bits is that splitting out the linker also makes more of the build cacheable</p>



<a name="204265923"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Linker%20pipelining/near/204265923" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Linker.20pipelining.html#204265923">(Jul 17 2020 at 22:15)</a>:</h4>
<p>b/c we basically just need to cache the rlib just-before-the-linker</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>